Conversation
There was a problem hiding this comment.
Pull request overview
This PR updates Apache DataFusion from v50.2.0 to v51.0.0 and Delta Lake from v0.29.4 to v0.30.0, along with several other dependent packages. The changes address all known breaking changes introduced by these new versions, ensuring compatibility while maintaining existing functionality.
Key Changes:
- Updated Arrow Flight client connection pattern to use explicit Endpoint creation
- Migrated from deprecated DeltaOps API to direct DeltaTable methods
- Updated IPC encoding API to include CompressionContext parameter
- Added new CREATE TABLE fields and unsupported feature checks for SQL parser compatibility
Reviewed changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| Cargo.toml | Updates core dependencies: Arrow 56.2.0→57.0.0, DataFusion 50.2.0→51.0.0, Delta Lake 0.29.4→0.30.0, and related packages |
| Cargo.lock | Comprehensive dependency lock file updates reflecting all transitive dependency changes |
| crates/modelardb_storage/src/parser.rs | Adds 6 new unsupported CREATE TABLE fields (dynamic, version, target_lag, warehouse, refresh_mode, initialize, require_user) for SQL parser compatibility |
| crates/modelardb_storage/src/optimizer/model_simple_aggregates.rs | Updates FileSource import and replaces predicate() with filter() method call |
| crates/modelardb_storage/src/lib.rs | Updates schema handling from Schema::from() to schema().inner().clone() pattern |
| crates/modelardb_storage/src/data_folder/mod.rs | Removes DeltaOps wrapper methods, updates SortingColumn construction from new() to struct literal, changes SortingColumn import from datafusion to deltalake |
| crates/modelardb_storage/src/data_folder/cluster.rs | Replaces metadata_delta_ops() with metadata_delta_table() for consistency |
| crates/modelardb_server/tests/integration_test.rs | Updates IPC encoding API: replaces encoded_batch() with encode() method and adds CompressionContext parameter |
| crates/modelardb_server/src/storage/data_transfer.rs | Replaces delta_ops().load() with delta_table().scan_table() API call |
| crates/modelardb_server/src/remote.rs | Updates Flight client connection to use Endpoint::new().connect() pattern and adds CompressionContext to IPC encoding |
| crates/modelardb_server/src/cluster.rs | Updates Flight client connection to use explicit Endpoint creation pattern |
| crates/modelardb_embedded/src/operations/data_folder.rs | Updates delta_ops().load() to delta_table().scan_table() and adjusts error message assertions for new DataFusion error format |
| crates/modelardb_embedded/src/operations/client.rs | Updates Flight client connection to use Endpoint::new().connect() pattern |
| crates/modelardb_client/src/main.rs | Updates Flight client connection to use explicit Endpoint creation pattern |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
This comment preserves information about the deadlock in |
This PR primarily updates Delta Lake to v0.30.0 and Apache DataFusion to v51.0.0 and fixes all known breaking changes. New versions of Apache DataFusion usually have a few breaking changes, so updating to each new release ensures that the number of changes required does not accumulate.